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Abstract 

We consider Markov chains on partially ordered sets that generalize 
the success-runs and remaining life chains in reliability theory. We find 
conditions for recurrence and transience and give simple expressions for 
the invariant distributions. We study a number of special cases, including 
rooted trees, uniform posets, and posets associated with positive semi- 
groups. 
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1 Partially Ordered Sets 
1.1 Preliminaries 

Suppose that {S, is a discrete partially ordered set. Recall that C C 5 is a 
chain if C is totally ordered under ^. We make the following assumptions: 

1. There is a minimum element e. 

2. For every x G S', every chain in S from e to a; is finite. 

Recall that y covers a; if ?; is a minimal element oi {t ^ S : t x}. The 
covering graph (or Hasse graph) of {S, ^) is the directed graph with vertex set 
S and edge set E = {{x, y) £ S'^ : y covers x}. From the assumptions, it follows 
that for each x ^ S, there is a (directed) path from e to x in the graph, and 
every such path is finite. For x G S, let 

— {y & S : y covers x}, = {w G S : x covers w} 

That is. Ax is the set of elements immediately after x in the partial order, while 
Bx is the set of elements immediately before x in the partial order. Note that Ax 
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could be empty or infinite. On the other hand, — 0, but for x ^ e, ^ % 
since there is at least one path from e to x. 

An upward run chain on (5*, ^) is a Markov chain that, at each transition, 
moves to a state immediately above the current state or back to e, the minimum 
state. A downward run chain is a Markov chain that, at each transition, moves 
to a state immediately below the current state, unless the current state is e in 
which case the chain can move anywhere in S. For particular posets, upward 
and downward runs can have applications in reliability theory, communications 
theory, queuing theory and other areas. Generally, posets are the natural math- 
ematical home for these stochastic processes. In this article, we are interested 
in general issues of recurrence, invariant distributions, time reversal, and results 
for special types of posets. See [I] for another class of Markov chains on posets. 

1.2 Uniform posets 

An interesting case is when the partially ordered set (S*, ^) is uniform in the 
sense that for each x € S, all paths from e to a; have the same length. It then 
follows that the all paths from x to y have the same length for any x, y G S 
with X ^ y; we denote this length by d(x,y). Let Sn = {x & S : d{e,x) — n} 
for n e N. Of course. So = {e} and {5'„ : n £ N} partitions S. 

1.3 Rooted trees and path space 

Another important special case is when the covering graph of (S*, ^) is a rooted 
tree with root e. In this case, Bx has a single element, which we will denote by 
x~ , for each x ^ e. There is a unique path from e to x for each x d S. Thus 
the poset {S, :<) is uniform, so the definitions in Section FOl applv. 

In fact, rooted trees form an essential special case, because we will show 
that upward and downward runs on an arbitrary poset can be constructed from 
upward and downward runs on a certain rooted tree of paths. Specifically, 
suppose that {S, ^) is a poset, and let S denote the set of finite, directed paths 
in 5*, starting at e. We define the partial order ^ on 5* by a ^ 6 if and only 
if a is a prefix of b. The covering graph of {S, :<) is a tree rooted at e (the 
degenerate path consisting only of e). For a G S, let 771(0) denote the endpoint 
of a. If y covers x in S then for every a e 5* with m{a) = x, ay covers a 
(where ay denotes that path obtained by appending y to the end of a). Let 
S{x) = {a £ S : m{a) = a;}, the set of paths ending in x. Note that for a 7^ e, 
a~ is the path obtained by removing the endpoint of a. If a £ 5*, the unique 
path from e to a in S simply consists of the successive prefixes of a. Thus in 
the notation of Section II. 2[ (i(e, a) is the length of the path a and hence S'„ is 
the set of paths of length n. 

1.4 Positive semigroups 

Another important special case is when the partially ordered set {S, ^) is as- 
sociated with a positive semigroup (S,-). That is, • is an associative binary 
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operation on S with an identity element e, no non-trivial inverses, and satis- 
fying the left-cancellation law. In this case, x ^ y if and only if there exists 
t G S with xt = y. Positive semigroups are essentially characterized by the 
fact that xS — {y ^ S : y > x} \s order-isomorphic to 5* for each x € S\ the 
mapping t ^-^ xt is an isomorphism. Assumption [T] is always satisfied since e is 
the minimum element. We will assume that \e,x\ = {t ^ S : t ^ x} is finite for 
each X G 5, so Assumption [2] is satisfied as well. Probability distributions on 
positive semigroups are studied in [H |31 IH [3 E] 

An element i £ S is irreducible if i cannot be factored, except for the trivial 
factoring i = ie = ei. If / is the set of irreducible elements of {S,-) then 
■A-x = {xi '■ i G /}, so in particular, ^{A^) is the same for each x. The poset 
{S, ^) will be uniform if and only if for every x € S, all factorings of x over I 
have the same number of factors. In this case, d{e, x) is the number of factors. 

The path space associated with (5, •) is isomorphic to the free semigroup 
on the set of irreducible elements /. This is the set of finite "words" over the 
alphabet /, with concatenation as the semigroup operation. That is, a path 
from e to a; in path space is uniquely associated with a factoring of x over I: 
X = iii2 ■ ■ - in] the string on the right is a word in the free semigroup. 



1.5 Probability Distributions 

Suppose that (5, ^) is a poset and that X is a random variable with support S. 
As usual, the probability density function (PDF) of X is the function / given 
by f{x) — P(X = x) for X IE S. The upper probability function (UPF) of X is 
the function F given by 

F{x)=¥{Xhx) = Y,fiy), 2; £5 
Finally, the rate function of X is the function r given by 

r(a;) = ^^=P(X = x|X>ra;), xeS 
F{x) 

In particular, X has constant rate if r is constant on S. For general posets, 
the distribution of X is not uniquely determined by the UPF F (and certainly 
not by the rate function r). These issues and the existence of constant rate 
distributions are explored in [7 . A special case of a general expected value 
result in is 

Y,F{x)=EmD[x])] (1) 
xes 

where D[x] = {t e x : t ^ x}. If (5, ^) is a rooted tree, then the UPF F of X 
does determine the distribution of X, since clearly the PDF / of X is given by 

fix) ^ Fix)- J2 p(y) 

Moreover, when (S*, <) is a rooted tree, #(-D[x]) = 1 + die,X) 
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2 The Upward Run Chain 



2.1 Basic definitions and results 

A Markov chain X = {Xq,Xi, . . .) on a poset {S, -<) is an upward run chain if 
the transition function P satisfies P{x, y) > if and only if y € or y = e. Of 
course, we must have 

P{x,e) = 1- P{x,A^), xeS 

Thus, in state x G S, the chain next moves to a state y G A^, oi back down 
to e. The chain is irreducible since e leads to every state, and every state leads 
back to e. The chain is aperiodic since e leads back to e in one step. If the state 
space is (N, <), then an upward run chain is simply a success-runs chain; these 
are commonly studied in reliability theory. 

For x € S, let denote the first positive hitting time to x: 

Tx = min{n G N+ : X„ — x} 

Define F : S {0,1] by F{x) = FeiT^ < T^). Of course, Pe(T^ = Te) = if 
x^e. Thus, F{x) = f^{T.^ <T^)iix^e while F{e) = 1. 

Proposition 1. Suppose that X is recurrent. Then F is left-invariant for P. 

Proof. Suppose first that y ^ e. Starting at e, the chain moves upward to y 
without an intermediate return to e if and only if the chain moves upward to 
some X E By, without an intermediate return to e, and then moves in one step 
from X to y. Therefore 

F{y)= E F{x)P{x,y) = {FP){y) 

xeB{y) 

On the other hand, to return to e, starting at e the chain must go directly back 
to e or move upward to some intermediate state x E S and then go back to e in 
one step. Thus, since the chain is recurrent, 

(FP)(e) = E F{x)P{x, e) = Pe(Te < oo) = 1 = F(e) 

□ 

We will refer to F as the standard invariant function for X. In the recurrent 
case, the chain is positive recurrent if and only if 

M(e):=E^(^) (2) 
xes 

is finite. In this case, the invariant PDF / is given by f{x) = F(x)/ii{e) for 
x & S. Equivalently, ^x{x) := Ea;(Tj,) = ii{e)/F{x) for x G S. 
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2.2 Upward runs on rooted trees 

Consider now the special case where the covering graph of {S, :<) is a rooted 
tree. If x G S and exiX2 ■ ■ • Xn-i x is the unique path in S from e to x, then 
clearly 

F{x) ^ Pie,xi)P{xi,X2) ■ ■ ■ P{xn-ux) (3) 

the product of the transition probabilities along the path from e to x. It follows 
that 

Pe(Te > n) = ^ F{x), neN (4) 

Thus the chain is recurrent if and only if the sum in ^ has limit as n — >■ cxd. 

Suppose that the chain is recurrent. Then clearly F is the UPF of Xx(e)-i, 
the last state vistied before returning to e (starting at e). That is 

F{x) = Pe(r, < Te) = Pe(XT(e)-l h x) , X^S 

Note from ^ that /i(e) = Ee(Te) can be written as 

Me)-1+Ee [d (e,XT(e)-i)] 

In the positive recurrent case, the invariant PDF / is the function obtained by 
normalizing F with /i(e). Note that in general, / is not the PDF of Xx(e)-i) 
given Xq — e. In fact, the two PDFs are the same if and only if the invariant 
distribution has constant rate. 

Conversely, given an UPF F on S, we can construct a recurrent upward run 
chain with F as the standard invariant function. Specifically, suppose that X 
is random variable with support S and with UPF F and PDF /. Define P by 

P{x,y)^¥iXhy\Xhx) 

P{x,e) ^¥{X = x\Xhx) 

Then P is the transition probability function for an upward run chain X = 
{Xq,Xi, . . .). Moreover, F and P satisfy ([3]), so the notation is consistent and 
F is the standard invariant function. Note that x i— >■ P(x, e) is the rate function 
of X. In particular, if P{x, e) = a for all x 6 5', then X has constant rate a, 
the chain is positive recurrent, and / is the invariant PDF. 

2.3 Upward runs on path space 

Now return to the general case where (5, ^) is an arbitrary poset and (5, <) is 
the corresponding path space discussed in Section [1.31 (a rooted tree). Suppose 
that P is the transition probability function of an upward run chain on S. We 
define P on 5' by 

P(a, ay) = P{x, y), P(a, e) = P(a;, e), a € S{x), x € S 



Fjy) 
F{xy 

fix) 
Fix)' 



X e S, y e 
x(£S 
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Clearly P is a valid transition probability function and corresponds to an upward 
run chain on 5*. Moreover, starting at e, we can define the two chains on a 
common probability space. To do this, we need some notation. If is a 
process on S, define the process X on S" by X„ = m{Xn). 

Proposition 2. If X is an upward run chain on S with transition probability 
function P and starting at e, then X is an upward run chain on S with transition 
probability function P and starting at e. 

Proof. The proof follows easily from the definition of the path space S and the 
fact that 

P(a, b) = P(m(a), m{b)), a, b e S 

□ 

Of course the results of Section l2.2l applv to X. Also, with the chains coupled 
as in Proposition [21 note that Tg — Tg. If a ~ e xi X2 ■ ■ ■ x„ G S then from (j4]) 
and the definition of P, 

F{a) = P{e, exi)P{exi, ex-yx^) ■ ■ ■ P{ex\ ■ • ■ ex\ • ■ • Xn) 

= P{e,Xi)P{xi,X2) ■ ■ ■ P{Xn-l,Xn) 

Now, for X to go from e to x without an intermediate return to e, X must 
move along some path from e to a; so 

F{x) = F{a), xeS 

a£S{x) 

It then follows that 

Pe(Te >n)=J2 ^(«) (5) 

oGS„ 

The chains X and X have the same classification: recurrent if and only if the 
sum in ([5]) has limit as n — oo, and of course positive recurrent if and only 
if the sum in ^ is finite. In the positive recurrent case, the invariant PDFs / 
and / are related by 

Since (5, is a rooted tree, F is an UPF if the chains are recurrent. On 
the other hand F may not be an UPF on S, and conversely, there may exist 
UPFs F on S that cannot be realized as the standard invariant function for a 
recurrent upward run chain on S. 

3 The Downward Run Chain 
3.1 Basic definitions and results 

A Markov chain Y = (Yq,Yi, • • • ) on (5 ^), is a downward run chain if the 
transition probability function Q satisfies Q{x, y) > if and only if x = e and 
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y G S', or if x G S' — {e} and y G Bx- Thus, the chain moves downward from 
a state to one of its predecessors, until it gets to e; then it can move anywhere 
in S. The chain is irreducible since every state leads to e and e leads to every 
state. The chain is aperiodic since e leads to e in one step. If the posct is 
(N, <), then a downward run chain is simply a remaining life chain; these are 
commonly studied in reliability theory. 

We denote the first positive hitting time to a; G 5 by 

Ux = min{n G N+ : y„ = a;} 

A downward chain is always recurrent. Since all paths from e to a; are finite, 
Px(?7e < oo) = 1 for a; G 5 - {e}. Hence 

Pe(f/e < oo) = 0(e, e) + Q(e,a;)P^(C/e < oo) = 1 

xeS-{e} 

Define G : 5 (0, 1] by G{x) = Pe(C4 < Ue)- Of course, Pe(?7x = U^) = Q 

if X 7^ e. Thus, G{x) = Pe(f/:r < f/e) if a; ^ e while G(e) = 1. Note that G 
is defined for the downward run chain just as F is defined for the upward run 
chain. 

Proposition 3. The function G is left-invariant for Q. 
Proof. For y € S, 

{GQ){y) = J2G{x)Q{x,y) 
xes 

= G{e)Q{e,y)+ ^ G{x)Q{x,y) 

xeA{y) 

= Q{e,y)+ G{x)Q{x,y) = G{y) 

xeA{y) 

For the last line, note that starting at e, the chain hits y before returning to 
e if and only if the chain jumps immediately to y or hits some x & Ay, before 
returning to e, and then moves from a; to y in one step. □ 

As before, we will refer to G as the standard invariant function for Y. The 
chain is positive recurrent if and only if 

v{e) :=^G(a;) 
xes 

is finite. In this case, the invariant PDF is given by g{x) = G{x)/v(e) for x € S. 
Equivalently, i/(a;) := Ex{Ux) = i'{e)/G{x) for x G S. 
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3.2 Downward runs on rooted trees 

Consider the special case where the covering graph of {S, :<) is a rooted tree. 
Since a non-root vertex has a single parent, we have 

Q{x, x~) = 1, X S — {e} 

Thus, the invariant function G is simply the UPF of Yi, given Yq = e: 

G(a;) = ^g(e,y)=Pe(ri xeS 

From ([Ij, vie) — Ee(C/e) can be written as 

v{e) = l + ]Ee[d(e,ri)] 

Note again that in the positive recurrent case, the invariant PDF is the function 
g obtained by normalizing G, and is not in general the PDF of Yi given Yq — e. 
The two PDFs are the same if and only if the distribution has constant rate. 

Conversely, given an UPF G on S, it's trivial to construct a downward run 
chain Y with G as the standard invariant function. Specifically, suppose that G 
is the UPF corresponding to the PDF g. We just need to define Q{x, e) = g{x) 
for X G S, and of course Q{x, x~) = 1 ioi x G S — {e}. 

3.3 Downward runs on path space 

Now return to the general case where (5, ^) is an arbitrary poset and {S, ^) is 
the corresponding path space discussed in Section [Ol fa rooted tree). Suppose 
that Q is the transition probability function of a downward run chain on S. We 
define Q on S* as follows: 

Q{e,exi ■ --Xn) = Q{e, x„)Q{x„, x„-i) ■ ■ •(3(a;i,e), exi - ■ ■ Xn & S 
Q{a,a^) = l, a G 5 — {e} 

It's easy to see Q is a valid transition probability function and corresponds to 
a downward run chain on S. Moreover, starting at e, we can define the two 
chains on a common probability space. As before, if Y' is a process on S, define 
the process Y on S by Yn = m(F„). The proof of the following proposition is 
straightforward . 

Proposition 4. If Y' is a downward run chain on S with transition probability 
function Q and starting at e, then Y" is a downward run chain on S with 
transition probability function Q and starting at e. 

Of course the results of Section [5^ applv to Y. Also, with the chains coupled 
as in Proposition [21 note that = Tg. The chains Y and Y have the same 
classification. 

Since {S, :<) is a rooted tree, G is an UPF. On the other hand G may not 
be an UPF on S, and conversely, there may exist UPFs G on S* that cannot be 
realized as the standard invariant function for a recurrent downward run chain 
on S. 
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4 Time Reversal 



The class of recurrent upward run chains and the class of downward run chains 
are time reversals of each other. 

4.1 Reversing an upward run chain 

Suppose that X is a recurrent upward run chain with the transition probability 
function P and standard invariant function F (and other notation) as in Section 
[21 Then the transition probability function Q associated with the time reversed 
chain satisfies 

Fiy)Q{y,x)^F{x)P{x,y), x, y e S 

Thus, Q{y,x) > if and only if y = e, or y ^ e and x G By. Hence the time 
reversed chain 1^ is a downward run chain and 

Q{y,x) = ^P{x,y) (6) 

Of course F is also invariant for Q and satisfies F{e) ~ 1 so F ^ G. In the 
positive recurrent case, f = g and ^ = v. 
We can interpret ([6]) as 

In particular, when y — e, we have 

Q{e,x)=Pe[XT(e)-l^x] 

Note that if P{x, e) is constant in x, then the chains are positive recurrent, 
f{x) = Q{e,x) is the invariant PDF. That is, 

Q{e,x)^^, xeS 
/i(e) 

4.2 Reversing a downward run chain 

Suppose that 1^ is a downward run chain with transition probability function 
Q and standard invariant function G (and other notation) as in Section [H The 
transition probability function P associated with the time reversal satisfies 

G{x)P{x,y)=G{y)Qiy,x), x, y e S 

Thus P{x, y) > if and only if y G oi y — e. Hence, the time reversed chain 
X is an upward run chain and 

P{x,y)^^Q{y,x) (7) 

As before, F = G, and in the positive recurrent case, f = ji = v. 
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We can interpret ([7]) as 

P(a;,y)-Pe[Ft/(,)_i=j/|C/. <C/e] 

When y — e, ^ gives 
which we can interpret as 

P{x,e) = Pe[C/, = 1|C/:. < Ue] = FeiYl = x\U^ < U,) 

If P{x, e) is constant in a: G 5 then from (|5]), the chains are positive recurrent 
and g{x) — Q{e,x) is the invariant PDF. 

5 Examples and Special Cases 
5.1 Uniform Posets 

Suppose that the poset (5, ^) is uniform, as defined in Section FOl The general 
resuhs in Sections [2] and [3] simplify significantly. 

For the upward run chain X = {Xq, Xi, . . .), note that 

F{x) ^ P'^^'''''\e,x), xeS 

That is, F{x) is the probability, starting at e, that the chain moves strictly 
upward in S, reaching state x in the minimum time d{e, x). Thus, 

^(x) = P"(e,a;), x e n e N 

FeiTe > n) = P"(e, S'„), n e N (9) 

CXj 

/z(e)=Ee(Te) = ^P"(e,^„) (10) 

The chain is recurrent if and only if the expression in ^ has limit as n cxi 
and positive recurrent if and only if the sum in (jlOp is finite. In the positive 
recurrent case, the invariant PDF / is given by 

P"fp x) 

fix) = y\ xeSn,neN 

Consider the downward run chain Y = {Yq, Yi,- ■ ■). If Fq = e then Ue = 
n + 1 if and only if Yi £ Sn ■ Hence 

Pe(C/e - n + 1) = Pe(ri £ Sn) = 0(e, Sn) 

and so 

oo 

Ke):=Ee(;7e) = ^(n + l)g(x,5„) (11) 

n=0 
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Thus the chain is positive recurrent if and only if the sum in PT|) is finite. The 
standard invariant function G also simplifies 

G(x)=^g(e,2/)g^(--^)(y,x), x^S 

Now, for the upward run chain X, consider the special case where 

P{x, Ax) = an, P{x, e) = 1 - a„; x ^ 5„, n e N 

where q;„ G (0, 1) for n G N. Thus, the chain moves from level n to level n + 1 
with probability q;„, and resets to e with probability 1 — an- Let P+(x,y) = 
P{x,y)/an for x £ Sn and let -/V„ = d(e,X„) for n € N. Then clearly 
AT = {Nq, N2, . . .) is an ordinary success-runs chain on (N, +) while Pj^ is the 
transition probability for a Markov chain that moves strictly upward in the 
poset. The behavior of the upward run chain X and the corresponding reversed 
downward run chain Y can be explained simply in terms of N and P+ . 
Of course, the transition matrix P for N is given by 

P(n, n + 1) = a„, P(n, 0) = 1 - a„, nSN 

Clearly Tq = where Tq is the hitting time to for the chain TV. The standard 
invariant function F for TV is 

F{n) = QfQ • • • a„_i, n e N 

Both chains are recurrent if and only if nfe°=o = ^ and both chains are positive 
recurrent if and only if /i(0) — X^^^o Q^o • • • Q^n-i < c>o. In the positive recurrent 
case, the invariant PDF for TV is 

Ai(0) 

Returning to the upward run chain X, note that the standard invariant 
function F satisfies 

F{x)^ F{n)P'l{e,x), x£Sn,nen 

In the positive recurrent case, the invariant PDF similarly satisfies 

/(x) = /(n)P;'(e,x), xe5„, neN 

The downward run chain obtained by reversing TV has transition probabili- 
ties 

Q{n -I- 1, n) = 1, (5(0, n) = ao ■ • • a„_i(l - q;„), n G N 
The downward run chain obtained by reversing X has transition probabilities 

, Pl{e,x)P+{x,y) 

Q{y,x) = — — — - — , x £ Sn,y ^ Ax,n 
^+ i^i y) 

Q(e,a;) = g(0,n)P;'(e,a;), xGS'„,neN 

Note in particular that for y )^ e and x G By the downward probability Q{y,x) 
is independent of the parameters (ao, ai, . . .). 



11 



5.2 Positive semigroups 

Suppose now that the poset {S, :<) is associated with a positive semigroup (5, •) 
with / as the set of irreducible elements. It's natural to consider upward and 
downward runs that take advantage of the self-similarity noted in Section 11.41 
An upward run chain X on {S, ■) is spatially homogeneous if P{x,xi) = r.i, 
for all X G S and i S / where > and r :— J2i£i'''i < 1- I* follows of 
course that P{x,e) = 1 — r for a; G 5. The standard invariant function for the 
corresponding chain on the free semigroup (/*, ■) is 

n 

F*{il i2 ■ ■ ■ in) = Y]_ '^iki («1,«2, ■ • • ,«n) e /" 
fe=l 

or equivalently F*{a) — Ilie/'""''""^ where ni{a) is the number of times that 
letter i appears in word a. The chain is positive recurrent and moreover, the 
invariant distribution is "exponential" (see |3]). The corresponding standard 
invariant function F for X is given by 

1-112- ■ -in—X 

Of course, these results are a special case of the construction in Section [2l since 
(/*, •) is isomorphic to the path space {S, :<). 

For the downward run chain Y obtained by reversing X, the function x i— >■ 
Q{e, x) is the invariant PDF /, since P{x, e) is constant in x. 

5.3 Upward and downward runs on (N'^,-!-) 

Consider the case of the uniform positive semigroup (N*^, +). For i = 1, . . . ,k, 
let Ui G N*"' be the element with 1 in position i and in all other positions; these 
are the irreducible elements. For x gN'', let 




This is a multinomial coefficient and gives the number of factorings of x over /; 
in each factoring, Ui must occur Xi times. 

Consider the upward run chain with uniform probabilities, as in Sections 
15.11 and 15.21 Specifically, let rt — P{x,x + Ui), independent of a; g N^, where 
r := X]i=iPi 1- For the upward run chain, the standard invariant function F 
is given by 

k 

F{x) ^ C{x)Y[r^\ xeN'' 

i=l 

The chain is positive recurrent and the invariant PDF is given by 

k 

fix) = (1 - r)F{x) = (1 - r)C{x) [| rf' , a; £ N'= 

i=l 



12 



li Z = (Zi, . . . , Zk) is a random vector with the invariant distribution then the 
(marginal) distribution of Z^ is geometric on N with rate parameter 

^ ^ n 

For the downward run chain Y obtained by reversing X, the downward 
probabihties are given by 

Qix,x~Ui)^ , a;GN-{0} 

Note that these probabihties are independent of (ri, r2, . . . , rfc). 
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